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(57) Abstract: The present invention provides 
a 3D handwriting recognition system that allows 
users to freely write words or characters in a 3D 
space in a touchless manner, without requiring any 
physical medium such as a pad or a tablet. The 
users' handwrting input in a 3D space will be tracked 
by an input device of the system that generates 
corresponding 3D motion data and wirelessly 
transfers the 3D motion data to a recognition 
device of the system. The 3D motion data will 
be converted and then mapped onto a 2D plane to 
generate corresponding 2D images for handwriting 
recognition. In this way, the users' inputting will 
never be limited to any screen, pad or plane, and the 
users will have more flexibility and enjoyable writing 
experience. 
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METHOD AND SYSTEM FOR THREE-DIMENSIONAL HANDWRITING 

RECOGNITION 

5 

BACKGROUND OF THE INVENTION 

The invention relates generally to handwriting recognition technologies, and 
more particularly to three-dimensional (3D) handwriting recognition methods and 
systems. 

10 

Handwriting recognition is a technique by which an intelligence system can 
recognize characters and other symbols written by hand. This technique has been 
very popular since it frees users from the keyboard, allowing users to write and draw in 
a more natural way. With the increasing demands from users, more and more devices 
15 now have incorporated handwriting recognition system to give users natural input 
experience. Handwriting recognition is particularly popular in various handheld 
devices that provides handwriting recognition of words with complex structures, e.g., 
Chinese characters, input of which using a conventional keyboard is very time 
consuming. 

20 

A handwriting input device provides a user with a friendly way of inputting 
information. At present, the minimum requirement for an inputting device is a mouse. 
To write with the mouse, the user needs to press and hold down the primary mouse 
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button, and then move the mouse pointer to form strokes of a word or character for 
generating the final word or character. Popular handwriting input devices, such as a 
pen stylus and tablet, are used on conventional handheld devices such as PDAs, or 
are connected to a computer through a serial or USB port. Handheld devices often 
5 use a pen stylus and a touch screen as a pad to allow users to perform recognition 
functions. Most handheld devices, such as PDAs, are equipped with this kind of input 
device. 

Another kind of handwriting input device includes a pen that allows users to 
10 transfer data into a receiving unit, such as a cellular phone, a PDA or a PC by simply 
writing or drawing in their own natural handwriting on the comfort and space of a 
regular piece of paper. 

At present, all conventional handwriting input devices adopt a two-dimensional 
15 input method. Users have to write on a physical medium, such as a tablet, a touch 
screen, or a paper pad. This limits the choices available for users. For example, if a 
user wants to write some comments during a presentation or a lecture, he or she 
would have to first find a physical medium, e.g., a table, a paper pad, etc. This can 
cause much inconvenience for the user while standing in the room, giving the 
20 presentation or lecture. Also, in a mobile environment, such as in a car, a bus or a 
subway, it would be very inconvenient to "write" on a physical medium using a stylus. 
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Therefore, there is a need to provide an improved handwriting recognition 
system that gives users more flexibility and convenience and frees the users from the 
physical medium required for two-dimensional handwriting recognition. 
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SUMMARY OF THE INVENTION 

The present invention gives users more flexibility and enjoyable writing 
experience by allowing users to freely write words or characters In a 3D space In a 
touchless way, without requiring any physical medium such as a pad or a tablet. 



5 



According to the invention, there is provided a handwriting recognition system 
that comprises an input device and a recognition device in communication with the 
input device. The input device includes a three-dimensional (3D) motion detection 
sensor configured to generate 3D motion data in response to a 3D motion. In one 
10 embodiment, the motion detection sensor measures acceleration of the 3D motion in 
X, Y and Z axial directions to generate the 3D motion data. The recognition device is 
configured to receive (e.g., by wireless means) the 3D motion data from the input 
device and derive corresponding two-dimensional (2D) images for handwriting 
recognition, based on the 3D motion data. 

15 According to one embodiment of the invention, the recognition device calculates 

corresponding 3D coordinates based on the 3D motion data, constructs corresponding 
3D tracks based the 3D coordinates, and derives the corresponding 2D images from 
the 3D tracks by mapping the 3D tracks onto a 2D plane for handwriting recognition. 

Other objects and attainments together with a fuller understanding of the 
20 invention will become apparent and appreciated by referring to the following 
description and claims taken in conjunction with the accompanying drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is explained in further detail, and by way of example, with 
reference to the accompanying drawings wherein: 

FIG. 1 shows a three-dimensional handwriting recognition system according to 
5 one embodiment of the invention; 

FIG. 2 is a flow chart diagram illustrating a recognition process according to one 
embodiment of the invention; 

FIG. 3A shows a 2D image of a Chinese character derived by mapping 3D 
tracks onto a 2D projection plane; 

0 FIG. 3B shows a final result of handwriting recognition process based on the 2D 

image in FIG. 3A; 

FIG. 4 shows an external design of a 3D handwriting input device according to 
one embodiment of the invention; and 

FIG. 5 illustrates how the input device may be mounted. 

5 Throughout the drawings, the same reference numerals indicate similar or 

corresponding features or functions. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

FIG. 1 shows a three-dimensional handwriting recognition system 10 according 
to one embodiment of the invention. As illustrated, system 10 comprises a handwriting 
input device 20, a recognition device 30 and an output device 40. Input device 20 
includes a 3D motion detection sensor 22, a control circuit 26 and a communication 
interface 28. Recognition device 30 includes a processor 32, a memory 34, a storage 
device 36, and a communication interface 38. For simplicity, other conventional 
elements are not shown in FIG. 1. 



10 



In operation, a user moves input device 20 to freely write words or characters in 
a 3D space, e.g., in the air. Motion detection sensor 22 detects the 3D motion and 
communicates the 3D motion data and a sampling rate to recognition device 30 for 
handwriting recognition via a communication interface 28, such as Bluetooth, Zigbee, 
IEEE 802.11, infrared, or a USB port. The sampling rate may be a predetermined 
15 value set by an end user or a manufacturer based on factors such as the processing 
capability of the system. Alternatively, the sampling rate may be dynamically 
determined and adjusted based on, for example, the speed of the motion. This can be 
done, for example, by first determining the speed of the initial motion associated with 
the handwriting. Then the recognition device can dynamically adjust the sampling rate 
20 based on the speed of the motion at the last sampling point. The higher the speed is, 
the higher the sampling rate will be, and vice versa. By dynamically adjusting the 
sampling rate, the accuracy of the handwriting recognition can be improved since only 
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the optimal number of sampling points will be used for constructing the word or 
character. Further, lower power consumption is needed. 

Based on the motion data and the sampling rate received from input device 20, 
5 processor 32 calculates the corresponding 3D coordinates on the X, Y and Z axes and 
stores these coordinates in storage device 36. Using the calculated coordinates, 
processor 32 constructs the corresponding 3D tracks. The 3D tracks will then be 
projected onto a 2D plane to form 2D images which will be recognized using 
conventional handwriting recognition software. The final result is displayed on output 
10 device 40. 

Since 3D writing is a continuous process, control circuit 26 of input device 20 
provides a control signal to recognition device 30 via interface 28 to indicate 
separation of individual words or characters upon receiving a user provided external 
15 input. For example, the user may press a control button to cause control circuit 26 to 
generate the control signal after completion of writing a word or a character. 

According to a specific embodiment of the invention, motion detection sensor 
22 detects the 3D motion by measuring the acceleration of the movement along the X, 
20 Y and Z axes. As an example, the piezoresistive-type tri-axial accelerating sensor 
commercially available from Hitachi Metals, Ltd., Tokyo, Japan, may be used as 
motion detection sensor 22. This accelerating sensor in the form of an IC chip has the 
ability to simultaneously detect acceleration in the three axial directions (X, Y and Z). 

7 
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The senor is highly sensitive and shock resistant and is a very small and thin 
semiconductor type 3 axial accelerating sensor. More information about this 
accelerating sensor is available on the following website http^/www.hitachi- 
metals.co.jp/e/prod/prod06/p06_10.html, which is hereby incorporated by reference. 

5 

FIG. 2 is a flow chart diagram illustrating a recognition process 100 performed 
by recognition device 30, according to one embodiment of the invention. In FIG. 2, 
recognition device 30 receives the 3D motion data (e.g., the acceleration data of the 
movement in the X, Y and Z directions) and the sampling rate from input device 20 
10 (step 102). Based on the information received, processor 32 calculates the 
corresponding 3D coordinates on the X, Y and Z axes for each sampling point using 
the starting point of the movement as the origin (step 106). Each sampling point is 
also used as a reference point for calculating the coordinates of the following sampling 
point. 

15 

Calculation of the 3D coordinates is continuously performed based on the 
incoming 3D motion data until processor 32 detects receipt of a control signal (step 
112). The control signal indicates completion of writing a word or a character. The 
corresponding 3D tracks are constructed using the 3D coordinates (step 116) and are 
20 then mapped onto a 2D plane (step 122). Thereafter, conventional 2D handwriting 
recognition is performed (step 1 26). 



8 



BNSDOCID: <WO 2004029B66A1 J_> 



WO 2004/029866 



PCT/IB2003/004102 



In the above recognition process 100, to map the 3D tracks onto a 2D plane at 
step 122, it is necessary to first find a proper 2D projection plane. In a preferred 
embodiment of the invention, a proper 2D projection plane is separately derived for 
each word or character. 

5 

According to the geometry principles, a proper 2D projection plane is a plane, to 
which the sum of the distance square of each sampling point is minimal. Assume the 
coordinates of n sampling points are known as follows: 
( x \ 9 yx>zMx 29 y 2i z 2 )...(x n9 y nt z n ) , and the equation of the plane is 

10 Ax+By+Cz+D=0, where *0) . Now it is necessary to determine the 

values of A,B,C,D. The distance from one point (x^,,*,) to the plane is given as 

d x =■ — ' 1 1 , and the sum Yd 2 represented by F(A,B,C,D) is 

y/A 2 +B 2 +C 2 m 

expressed as: 

FXAB CD>=Yd 2 = ^ +B)i ^ +D)2 +( ^ +B % +C! * +D ? ±z*i£k *g& +Q i +D ? 
J A 2 +B 2 +C 

15 

The values of A,B,C,D can be determined using the Lagrange multiplication 
method as follows, which is described in Mathematics Analysis by Ouyang 
Guangzhong, published by Fudan University Press in 1999 in China, which is hereby 
incorporated by reference. Thus, 

20 F(A,B,C,D) = F'(A,B,C,D) = (Ax l + By t + Cz, +D) 2 +(.4x 2 + By 2 +Cz 1 +Df +...+(Ax, +By a + Cz„ +D) 2 
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10 



15 



under the constraint - l: From mis equation, the Wowing equation is 

derived: 

G(4B,C,D) = F'(A,B,C,D) + Z(A 2 +B 2 + C 2 -l) , 
where A is a Lagrange multiplier, which is a constant. The partial differential functions 
5 are now performed on G(A > B,C,D) with respect to A, B, C and D as follows: 

BA 

dG{A,B,C,D) 

as U 

dG{A,B,C,D) 
3C 

dG(A,B,C,D) _ 
dD 

From the above four equations, the following equations are obtained: 

^E.^*J.)+^OiV,)+^)+CE ( 0' / ^) + 2)2^j il .o 

^L>. +5 E>, + +nD = 0 

^ 2 +S J +C 2 =l 
where equation (4) can be rewritten as: 

By incorporating equation (6) into equations (1), (2), and (3), the following is 
obtained: 

10 
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" 71 71 

ZLcs^—SL^*^) 2L<ft*»>4S.fe*») E«k*s)-iE«<**«) 



A 



(7) 



Thus, from the above equations, the values of A, B, C and D can be obtained. 



As an example, the following 3D coordinates for a total of 13 sampling points 
5 are obtained for a Chinese character 

{(0,0,0),(0.49,-1 ,0.02),(1, 0,0.03),(0,0,0.02),(2,-0.1 3,0.01 ),(3.5,-0.14,0),(3.76,-0.31, - 
0.01 ),(2.74,-1. 26,0.01 ),(1.38,-2.24,0),(2.5,-2,0.01),(1. 746,1, 0.02, ),(2,-1.5,0.03),(1. 876,- 
4.5,0.02),}. 



10 By using the Lagrange multiplication method described above, the following are 
obtained: 



" 53.407 


-23.6725 


0.1911 " 




~A 




~A 


-23.6725 


36.2195 


-0.2084 


* 


B 


= -X 


B 


0.1911 


-0.2084 


0.0035 _ 




C 




C 



A 2 +B 2 +C 2 =l,and 

15 From the above equations, the values of A, B, C and D are determined as follows: 
A = 0.0045,5 = 0.0023.C = 0.9999,0 = -0.01777 . 

Thus, the equation of the 2D projection plane is 
0.0045*+0.0023;y + 0.9999z-0.01777 = 0. From the equation of the projection plane, 

11 
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Ax+By+Cz+D = 0, and the equation of a line that is vertical to the projection plane, 

x — x, y-y- z — z- 

—j- = = —jr- ■ tne following equations are derived: 

x _ (B 2 + C 2 )x, - A(By, + Cz i +D) 
A 2 +B 2 +C 2 

_ (A 2 + C 2 )y, - B(Ax, + Cz, + D) 
A 2 +B 2 +C 2 

5 which can be used to obtain the corresponding 2D coordinates for each 3D sampling 
point. In this example, the following corresponding 2D coordinates are obtained: 

{(0.0001 ) 0) I (0.49 I -1),(1,.2.0001),(0,0),(2 1 -0.13),(3.5001 I -0.14),(3.7601,-0.31) I (2.7401,- 
1 .26),(1 .3801 ,2.24),(2.5001 ,-2),(1 .746,1 ),(2,-1 .5001 ),(1 .876,-4.5001 )}. 
Based on these 2D coordinates, the corresponding 2D image is projected onto the 2D 
10 projection plane. The result of this Chinese character is shown in FIG. 3A. The final 
result is generated by performing a conventional 2D handwriting recognition process, 
as shown in FIG. 3B. 

In the above, the Lagrange multiplication method is used to obtain the values of 
15 A, B, C and D. Other methods such as linear regression method may also be used. 

FIG. 4 shows an external design of a 3D handwriting input device 200 
according to one embodiment of the invention. As shown in FIG. 4, input device 200 
includes a housing 210 that contains the electronics parts of the device (such as a 3D 
20 motion detection sensor IC chip), a control button 212 for allowing a user to input a 
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control signal to indicate completion of writing a word or character, and a band 220 for 
mounting input device 200 on the user's finger. 

FIG. 5 illustrates how input device 200 may be mounted. In FIG. 5, device 200 
is mounted on a finger 232 of a user's hand 230. By mounting it on the finger, the user 
can simply move the finger to write any word or character in a 3D space. The 3D 
motion data will be wirelessly transmitted to a recognition device for handwriting 
recognition. 

According to the invention, the input device and the recognition device can be 
integrated together as a single unit (e.g., a pen) that operates in the same manner as 
described above. The final recognition results will be transmitted to an intelligent 
apparatus such as a PDA, a laptop computer, a PC, etc. 

The invention can also be used on a 2D plane if the user so chooses. In such a 
case, the coordinates calculated will be for 2D plane and the step of mapping the 3D 
tracks onto a 2D plane is omitted. 

While the invention has been described in conjunction with specific 
embodiments, it is evident that many alternatives, modifications and variations will be 
apparent to those skilled in the art in light of the foregoing description. Accordingly, it 
is intended to embrace all such alternatives, modifications and variations as fall within 
the spirit and scope of the appended claims. 

13 
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WHAT IS CLAIMED IS: 

1 . A handwriting recognition system, comprising: 

an input device including a three-dimensional (3D) motion detection sensor that 
is configured to generate 3D motion data in response to a 3D motion; and 

5 a recognition device, in communication with the input device, that is configured 

to receive the 3D motion data and derive corresponding two-dimensional (2D) images 
for handwriting recognition, based on the 3D motion data. 

2. The system of claim 1, wherein the recognition device includes means 
for performing 2D handwriting recognition based on the 2D images. 

10 3. The system of claim 1 , wherein the recognition device includes: 

means for calculating corresponding 3D coordinates based on the 3D 

motion data; 

means for constructing corresponding 3D tracks based the 3D 
coordinates; and 



15 



means for deriving the corresponding 2D images from the 3D tracks. 
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4. The system of claim 3, wherein the deriving means includes means for 
mapping the 3D tracks onto a 2D plane for deriving the 2D images for handwriting 
recognition. 

5. The system of claim 3, wherein the recognition device includes means 
5 for performing 2D handwriting recognition based on the 2D images. 

6. The system of claim 4, wherein the calculating means calculates the 
corresponding 3D coordinates of each sampling point based on the 3D motion data 
and a selected sampling rate. 

7. The system of claim 6, wherein the recognition device further includes 
10 means for dynamically adjusting the sampling rate based on the speed of the motion. 

8. The system of claim 6, wherein the deriving means includes means for 
deriving the 2D plane as a plane to which the sum of the distance square of each 
sampling point is minimal. 

9. The system of claim 3, wherein the Input device further includes a control 

15 circuit, responsive a user's command, that is configured to generate a control signal 

for transmitting to the recognition device to indicate completion of writing a word or a 
character. 

10. The system of claim 3, wherein the motion detection sensor measures 
acceleration of the 3D motion in X, Y and Z axial directions to generate the 3D motion 

20 data. 

15 

BNSDOCID- <W O__axWB9866A 1_l > 



WO 2004/029866 PCT/IB2003/004102 



11. The system of claim 5, further comprising an output device for displaying 
final results of the handwriting recognition. 

12. The system of claim 1, wherein the input device further includes a control 

circuit, responsive a user's command, that is configured to generate a control signal 

5 for transmitting to the recognition device to indicate completion of writing a word or a 
character. 

13. The system of claim 1, wherein the motion detection sensor measures 
acceleration of the 3D motion in X, Y and Z axial directions to generate the 3D motion 
data. 

10 14. The system of claim 1 , wherein the input device wirelessly transmits the 

3D motion data to the recognition device. 

15. The system of claim 1, wherein the recognition device includes means 
for performing 2D handwriting recognition based on the 2D images. 

16. A computing system, comprising: 
15 a memory; 

an input device including a three-dimensional (3D) motion detection sensor that 
is configured to generate 3D motion data in response to a 3D motion; and 
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a recognition device, operably coupled to the memory and in communication 
with the input device, that is configured to receive the 3D motion data and derive 
corresponding two-dimensional (2D) images for handwriting recognition, based on the 
3D motion data. 

5 17. The system of claim 16, wherein the recognition device includes means 

for performing 2D handwriting recognition based on the 2D images. 

18. The system of claim 16, wherein the recognition device includes: 
means for calculating corresponding 3D coordinates based on the 3D 

motion data; 

10 means for constructing corresponding 3D tracks based the 3D 

coordinates; and 

means for deriving the corresponding 2D images from the 3D tracks. 

19. The system of claim 18, wherein the deriving means includes means for 
mapping the 3D tracks onto a 2D plane for deriving the 2D images for handwriting 

15 recognition. 

20. A handwriting recognition method, comprising the steps of: 
generating 3D motion data in response to a 3D motion; and 

17 
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deriving corresponding two-dimensionai (2D) images for handwriting recognition 
based on the 3D motion data. 

21 . The method of claim 20, further comprising the step of performing 2D 
handwriting recognition based on the 2D images. 

5 22. The method of claim 20, further comprising the steps of: 

calculating corresponding 3D coordinates based on the 3D motion data; 
constructing corresponding 3D tracks based the 3D coordinates; and 
deriving the corresponding 2D images from the 3D tracks. 

23. The method of claim 22, wherein the step of deriving includes mapping 
10 the 3D tracks onto a 2D plane for deriving the 2D images for handwriting recognition. 

24. The method of Cairn 22, further comprising a step of performing a 2D 
handwriting recognition based on the 2D images. 

25. The method of claim 23, wherein the corresponding 3D coordinates of 
each sampling point are calculated based on the 3D motion data and a selected 

15 sampling rate. 

26. The method of claim 25, further comprising a step of dynamically 
adjusting the sampling rate based on the speed of the motion. 
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27. The method of claim 25, wherein the step of deriving further includes the 
step of deriving the 2D plane as a plane to which the sum of the distance square of 
each sampling point is minimal. 

28. The method of claim 22, further comprising the step of generating an 
5 indication, based on a user's command, to indicate completion of writing a word or a 

character. 

29. The method of claim 22, further comprising the step of measuring 
acceleration of the 3D motion in X, Y and Z axial directions and wherein the 3D motion 
data are generated based on the acceleration of the 3D motion in the X, Y and Z axial 

10 directions. 

30. The method of claim 24, further comprising a step of displaying final 
results of the handwriting recognition. 

31. The method of claim 20, further comprising the steps of wirelessly 
transmitting the 3D motion data and wirelessly receiving the 3D motion data for 

15 calculating the 3D coordinates. 

32. The method of claim 20, further comprising the step of measuring 
acceleration of the 3D motion in X, Y and Z axial directions and wherein the 3D motion 
data are generated based on the acceleration of the 3D motion in the X, Y and Z axial 
directions. 
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33. The method of oleim 20, further comprising the steps of tirelessly 
transmitting me 3D motion data and wirelessly receiving me 3D mot(oI) data for 
calculating the 3D coordinates. 

34. The method of claim 20, further comprising the step of performing 2D 
handwriting recognition based on the 2D images. 
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